Treat OSC ANSI Sequences as Invisible Text & Add OSC 8 Support #2544

eth-p · 2023-04-17T03:35:43Z

Fixes #2541 by ~~removing OSC sequences from lines before starting to print them.~~ no longer considering ANSI OSC sequences as visible text.


(Left: Fixed, Right: Current version)

Prior to this change, we don't interpret OSC sequences in any special way. They aren't caught by console::AnsiCodeIterator, so the OSC sequences were being treated as visible text.

When the OSC is printed in full, the terminal would see a valid sequence and hide it. As a consequence, lines would b e wrapped too early.
When the OSC is broken for line wrapping, the terminal wouldn't see a valid sequence. In this case, it prints both halves of the text on different lines and appears to be junk.

Edit: I had some extra time. A couple of new commits gave special handling for the OSC 8 sequence, allowing bat to disable the hyperlink at the end of the line and then re-emit it at the start of the next line.

eth-p · 2023-04-18T00:51:50Z

Made some changes so that we don't need to strip the sequences anymore. Instead, they'll be emitted as-is and just ignored for line wrapping purposes.

eth-p · 2023-04-18T01:04:54Z

I also found some possible optimizations while doing this:

The syntax highlighting style is being emitted for every single text chunk (rather than highlight region).

Essentially, for a string like Text^[33mMore Text^[1mEven More Text what we do is:
1. For Text
  - Emit highlight style: ^[38;2;100;100;100m
  - Emit text: Text
2. For ^[33m
  - Passthrough detected ANSI sequence: ^[33m
3. For More Text
  - Emit highlight style: ^[38;2;100;100;100m
  - Re-emit detected ANSI attributes as sequence: ^[33m
  - Emit text: More Text
4. For ^[1m
  - Passthrough detected ANSI sequence: ^[1m
5. For Even More Text
  - Emit highlight style: ^[38;2;100;100;100m
  - Re-emit detected ANSI attributes as sequence: ^[33m^[1m
  - Emit text: Even More Text
This gives us: ^[38;2;100;100;100mText^[33m^[38;2;100;100;100m^[33mMore Text^[1m^[38;2;100;100;100m^[33m^[1mEven More Text
Which could be simplified to: ^[38;2;100;100;100mText^[33mMore Text^[1mEven More Text

For the plain text syntax we don't actually have a style to emit (i.e. no ^[38;2;100;100;100m everywhere), but it still acts as though there is one and double-prints the passed through styles.
When wrapping is enabled, we could probably be reusing a single String as a line_buf.
As it is right now, we call String::with_capacity multiple times per line. A better approach would be to store it on the InteractivePrinter struct and reuse it across lines so we don't have to allocate as much.
I tested this, and it reduced performance on release builds. I guess reusing the variable prevents LLVM from performing a couple of optimizations.

eth-p · 2023-04-18T21:59:52Z

I figured with such a significant change to a hot loop, it might be worth benchmarking bat before and after my changes.

I no longer have an x86_64 system running MacOS to test with, but I can happily say that these changes actually result in a small performance gain under aarch64 :)

Notes

Both executables were compiled with cargo build --release --locked using rustc version 1.65.0.
bat-master is commit e828d78.
bat-with-fix is commit d899f82.
The lots-of-ansi.txt file was created using bat src/**/*.rs --decorations=always --wrap=never --paging=never --color=always > lots-of-ansi.txt, so it basically feeds bat's output back to itself.
Both versions output the same text, verified by running sha1sum against the the executables' output with the flags --terminal-width=80 --wrap=character --decorations=always --color=always lots-of-ansi.txt --paging=never

eth-p · 2024-02-08T01:09:05Z

Rebased to latest master and re-ran benchmarks.

Benchmarks: `aarch64`

Apple ARM M1 architecture, running Darwin 23.0.0 kernel.

With ANSI content in the input, these changes result in a minor performance increase.

With non-ANSI inputs, there were no performance regressions:

Benchmarks: `amd64`

Intel i7-4790k Haswell architecture, running Linux 6.1 kernel with glibc 2.38.

With ANSI content in the input, these changes result in a minor performance increase.

With non-ANSI inputs, there were no obvious performance regressions. Differences between the two builds was within margin of error.

If you would like benchmarks for a more modern architecture (AMD Zen 4) or for Windows, I can install some development tools on my other PC.

sharkdp

Thank you very much and sorry for leaving this hanging for so long.

Admittedly, I have not reviewed the very core of this (the escape sequence parser/iterator) in detail, but it looks very reasonable and is well covered by tests.

Also, thank you for performing the benchmarks!

eth-p · 2024-02-10T05:04:41Z

The failed audit check is related to libgit2, which isn't changed by this pull request. Merging this PR.

This can be used to extract a subset of ANSI escape sequences from a string of text. I have big plans for this eventually, but for now, it'll be used to strip OSC before printing.

This commit strips OSC (Operating System Command) sequences before printing lines. Eventually when time permits, I want to add back support for printing OSC sequences (and improve it to treat hyperlinks like an attribute). Until then, this should help prevent garbled output :)

More specifically, the test ensures that OSC sequences don't end up wrapping the line.

This is an iterator for escape sequences, using EscapeSequenceOffsetsIterator for the underlying parsing of individual escape sequences.

This uses the new EscapeSequenceIterator, saving us a preprocessing step for each line.

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [sharkdp/bat](https://github.com/sharkdp/bat) | minor | `v0.24.0` -> `v0.25.0` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>sharkdp/bat (sharkdp/bat)</summary> ### [`v0.25.0`](https://github.com/sharkdp/bat/blob/HEAD/CHANGELOG.md#v0250) [Compare Source](sharkdp/bat@v0.24.0...v0.25.0) #### Features - Set terminal title to file names when Paging is not Paging::Never [#2807](sharkdp/bat#2807) ([@Oliver-Looney](https://github.com/Oliver-Looney)) - `bat --squeeze-blank`/`bat -s` will now squeeze consecutive empty lines, see [#1441](sharkdp/bat#1441) ([@eth-p](https://github.com/eth-p)) and [#2665](sharkdp/bat#2665) ([@einfachIrgendwer0815](https://github.com/einfachIrgendwer0815)) - `bat --squeeze-limit` to set the maximum number of empty consecutive when using `--squeeze-blank`, see [#1441](sharkdp/bat#1441) ([@eth-p](https://github.com/eth-p)) and [#2665](sharkdp/bat#2665) ([@einfachIrgendwer0815](https://github.com/einfachIrgendwer0815)) - `PrettyPrinter::squeeze_empty_lines` to support line squeezing for bat as a library, see [#1441](sharkdp/bat#1441) ([@eth-p](https://github.com/eth-p)) and [#2665](sharkdp/bat#2665) ([@einfachIrgendwer0815](https://github.com/einfachIrgendwer0815)) - Syntax highlighting for JavaScript files that start with `#!/usr/bin/env bun` [#2913](sharkdp/bat#2913) ([@sharunkumar](https://github.com/sharunkumar)) - `bat --strip-ansi={never,always,auto}` to remove ANSI escape sequences from bat's input, see [#2999](sharkdp/bat#2999) ([@eth-p](https://github.com/eth-p)) - Add or remove individual style components without replacing all styles [#2929](sharkdp/bat#2929) ([@eth-p](https://github.com/eth-p)) - Automatically choose theme based on the terminal's color scheme, see [#2896](sharkdp/bat#2896) ([@bash](https://github.com/bash)) - Add option `--binary=as-text` for printing binary content, see issue [#2974](sharkdp/bat#2974) and MR [#2976](sharkdp/bat#2976) ([@einfachIrgendwer0815](https://github.com/einfachIrgendwer0815)) - Make shell completions available via `--completion <shell>`, see issue [#2057](sharkdp/bat#2057) and MR [#3126](sharkdp/bat#3126) ([@einfachIrgendwer0815](https://github.com/einfachIrgendwer0815)) - Syntax highlighting for puppet code blocks within Markdown files, see [#3152](sharkdp/bat#3152) ([@liliwilson](https://github.com/liliwilson)) #### Bugfixes - Fix long file name wrapping in header, see [#2835](sharkdp/bat#2835) ([@FilipRazek](https://github.com/FilipRazek)) - Fix `NO_COLOR` support, see [#2767](sharkdp/bat#2767) ([@acuteenvy](https://github.com/acuteenvy)) - Fix handling of inputs with OSC ANSI escape sequences, see [#2541](sharkdp/bat#2541) and [#2544](sharkdp/bat#2544) ([@eth-p](https://github.com/eth-p)) - Fix handling of inputs with combined ANSI color and attribute sequences, see [#2185](sharkdp/bat#2185) and [#2856](sharkdp/bat#2856) ([@eth-p](https://github.com/eth-p)) - Fix panel width when line 10000 wraps, see [#2854](sharkdp/bat#2854) ([@eth-p](https://github.com/eth-p)) - Fix compile issue of `time` dependency caused by standard library regression [#3045](sharkdp/bat#3045) ([@cyqsimon](https://github.com/cyqsimon)) - Fix override behavior of --plain and --paging, see issue [#2731](sharkdp/bat#2731) and MR [#3108](sharkdp/bat#3108) ([@einfachIrgendwer0815](https://github.com/einfachIrgendwer0815)) - Fix bugs in `$LESSOPEN` support, see [#2805](sharkdp/bat#2805) ([@Anomalocaridid](https://github.com/Anomalocaridid)) #### Other - Upgrade to Rust 2021 edition [#2748](sharkdp/bat#2748) ([@cyqsimon](https://github.com/cyqsimon)) - Refactor and cleanup build script [#2756](sharkdp/bat#2756) ([@cyqsimon](https://github.com/cyqsimon)) - Checks changelog has been written to for MRs in CI [#2766](sharkdp/bat#2766) ([@cyqsimon](https://github.com/cyqsimon)) - Use GitHub API to get correct MR submitter [#2791](sharkdp/bat#2791) ([@cyqsimon](https://github.com/cyqsimon)) - Minor benchmark script improvements [#2768](sharkdp/bat#2768) ([@cyqsimon](https://github.com/cyqsimon)) - Update Arch Linux package URL in README files [#2779](sharkdp/bat#2779) ([@brunobell](https://github.com/brunobell)) - Update and improve `zsh` completion, see [#2772](sharkdp/bat#2772) ([@okapia](https://github.com/okapia)) - More extensible syntax mapping mechanism [#2755](sharkdp/bat#2755) ([@cyqsimon](https://github.com/cyqsimon)) - Use proper Architecture for Debian packages built for musl, see [#2811](sharkdp/bat#2811) ([@Enselic](https://github.com/Enselic)) - Pull in fix for unsafe-libyaml security advisory, see [#2812](sharkdp/bat#2812) ([@dtolnay](https://github.com/dtolnay)) - Update git-version dependency to use Syn v2, see [#2816](sharkdp/bat#2816) ([@dtolnay](https://github.com/dtolnay)) - Update git2 dependency to v0.18.2, see [#2852](sharkdp/bat#2852) ([@eth-p](https://github.com/eth-p)) - Improve performance when color output disabled, see [#2397](sharkdp/bat#2397) and [#2857](sharkdp/bat#2857) ([@eth-p](https://github.com/eth-p)) - Relax syntax mapping rule restrictions to allow brace expansion [#2865](sharkdp/bat#2865) ([@cyqsimon](https://github.com/cyqsimon)) - Apply clippy fixes [#2864](sharkdp/bat#2864) ([@cyqsimon](https://github.com/cyqsimon)) - Faster startup by offloading glob matcher building to a worker thread [#2868](sharkdp/bat#2868) ([@cyqsimon](https://github.com/cyqsimon)) - Display which theme is the default one in basic output (no colors), see [#2937](sharkdp/bat#2937) ([@sblondon](https://github.com/sblondon)) - Display which theme is the default one in colored output, see [#2838](sharkdp/bat#2838) ([@sblondon](https://github.com/sblondon)) - Add aarch64-apple-darwin ("Apple Silicon") binary tarballs to releases, see [#2967](sharkdp/bat#2967) ([@someposer](https://github.com/someposer)) - Update the Lisp syntax, see [#2970](sharkdp/bat#2970) ([@ccqpein](https://github.com/ccqpein)) - Use bat's ANSI iterator during tab expansion, see [#2998](sharkdp/bat#2998) ([@eth-p](https://github.com/eth-p)) - Support 'statically linked binary' for aarch64 in 'Release' page, see [#2992](sharkdp/bat#2992) ([@tzq0301](https://github.com/tzq0301)) - Update options in shell completions and the man page of `bat`, see [#2995](sharkdp/bat#2995) ([@akinomyoga](https://github.com/akinomyoga)) - Update nix dev-dependency to v0.29.0, see [#3112](sharkdp/bat#3112) ([@decathorpe](https://github.com/decathorpe)) - Bump MSRV to [1.74](https://blog.rust-lang.org/2023/11/16/Rust-1.74.0.html), see [#3154](sharkdp/bat#3154) ([@keith-hall](https://github.com/keith-hall)) - Update clircle dependency to remove winapi transitive dependency, see [#3113](sharkdp/bat#3113) ([@niklasmohrin](https://github.com/niklasmohrin)) #### Syntaxes - `cmd-help`: scope subcommands followed by other terms, and other misc improvements, see [#2819](sharkdp/bat#2819) ([@victor-gp](https://github.com/victor-gp)) - Upgrade JQ syntax, see [#2820](sharkdp/bat#2820) ([@dependabot](https://github.com/dependabot)\[bot]) - Add syntax mapping for quadman quadlets [#2866](sharkdp/bat#2866) ([@cyqsimon](https://github.com/cyqsimon)) - Map containers .conf files to TOML syntax [#2867](sharkdp/bat#2867) ([@cyqsimon](https://github.com/cyqsimon)) - Associate `.xsh` files with `xonsh` syntax that is Python, see [#2840](sharkdp/bat#2840) ([@anki-code](https://github.com/anki-code)) - Associate JSON with Comments `.jsonc` with `json` syntax, see [#2795](sharkdp/bat#2795) ([@mxaddict](https://github.com/mxaddict)) - Associate JSON-LD `.jsonld` files with `json` syntax, see [#3037](sharkdp/bat#3037) ([@vorburger](https://github.com/vorburger)) - Associate `.textproto` files with `ProtoBuf` syntax, see [#3038](sharkdp/bat#3038) ([@vorburger](https://github.com/vorburger)) - Associate GeoJSON `.geojson` files with `json` syntax, see [#3084](sharkdp/bat#3084) ([@mvaaltola](https://github.com/mvaaltola)) - Associate `.aws/{config,credentials}`, see [#2795](sharkdp/bat#2795) ([@mxaddict](https://github.com/mxaddict)) - Associate Wireguard config `/etc/wireguard/*.conf`, see [#2874](sharkdp/bat#2874) ([@cyqsimon](https://github.com/cyqsimon)) - Add support for [CFML](https://www.adobe.com/products/coldfusion-family.html), see [#3031](sharkdp/bat#3031) ([@brenton-at-pieces](https://github.com/brenton-at-pieces)) - Map `*.mkd` files to `Markdown` syntax, see issue [#3060](sharkdp/bat#3060) and MR [#3061](sharkdp/bat#3061) ([@einfachIrgendwer0815](https://github.com/einfachIrgendwer0815)) - Add syntax mapping for CITATION.cff, see [#3103](sharkdp/bat#3103) ([@Ugzuzg](https://github.com/Ugzuzg)) - Add syntax mapping for kubernetes config files [#3049](sharkdp/bat#3049) ([@cyqsimon](https://github.com/cyqsimon)) - Adds support for pipe delimiter for CSV [#3115](sharkdp/bat#3115) ([@pratik-m](https://github.com/pratik-m)) - Add syntax mapping for `/etc/pacman.conf` [#2961](sharkdp/bat#2961) ([@cyqsimon](https://github.com/cyqsimon)) - Associate `uv.lock` with `TOML` syntax, see [#3132](sharkdp/bat#3132) ([@fepegar](https://github.com/fepegar)) #### Themes - Patched/improved themes for better Manpage syntax highlighting support, see [#2994](sharkdp/bat#2994) ([@keith-hall](https://github.com/keith-hall)). #### `bat` as a library - Changes to `syntax_mapping::SyntaxMapping` [#2755](sharkdp/bat#2755) ([@cyqsimon](https://github.com/cyqsimon)) - `SyntaxMapping::get_syntax_for` is now correctly public - \[BREAKING] `SyntaxMapping::{empty,builtin}` are removed; use `SyntaxMapping::new` instead - \[BREAKING] `SyntaxMapping::mappings` is replaced by `SyntaxMapping::{builtin,custom,all}_mappings` - Make `Controller::run_with_error_handler`'s error handler `FnMut`, see [#2831](sharkdp/bat#2831) ([@rhysd](https://github.com/rhysd)) - Improve compile time by 20%, see [#2815](sharkdp/bat#2815) ([@dtolnay](https://github.com/dtolnay)) - Add `theme::theme` for choosing an appropriate theme based on the terminal's color scheme, see [#2896](sharkdp/bat#2896) ([@bash](https://github.com/bash)) - \[BREAKING] Remove `HighlightingAssets::default_theme`. Use `theme::default_theme` instead. - Add `PrettyPrinter::print_with_writer` for custom output destinations, see [#3070](sharkdp/bat#3070) ([@kojix2](https://github.com/kojix2)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

eth-p force-pushed the fix-2541 branch from 5c90eca to c981895 Compare April 17, 2023 03:38

eth-p requested review from sharkdp and Enselic April 17, 2023 05:15

eth-p force-pushed the fix-2541 branch 3 times, most recently from a34f6f4 to bca3a3e Compare April 17, 2023 23:12

eth-p changed the title ~~Strip OSC Ansi Sequences Before Printing~~ Treat OSC ANSI Sequences as Invisible Text Apr 18, 2023

eth-p changed the title ~~Treat OSC ANSI Sequences as Invisible Text~~ Treat OSC ANSI Sequences as Invisible Text & Add OSC 8 Support Apr 18, 2023

eth-p force-pushed the fix-2541 branch from 782098a to d899f82 Compare April 18, 2023 02:23

eth-p mentioned this pull request Apr 18, 2023

Erratic output when OSC8 sequences are present #2541

Closed

eth-p mentioned this pull request May 23, 2023

Problem with ANSI escape color codes on Kitty (Fedora 36 on Sway) and Centos 8 Stream. #2568

Open

eth-p mentioned this pull request Jul 18, 2023

Support text with embedded ANSI codes for the higlight line #2621

Closed

keith-hall mentioned this pull request Oct 9, 2023

Multi line links get cut off when redirecting to them. #2702

Closed

eth-p force-pushed the fix-2541 branch from d899f82 to e6be1c5 Compare February 8, 2024 00:39

keith-hall approved these changes Feb 8, 2024

View reviewed changes

sharkdp approved these changes Feb 8, 2024

View reviewed changes

eth-p force-pushed the fix-2541 branch from f89c4c6 to 2f3bc7f Compare February 10, 2024 05:02

eth-p enabled auto-merge February 10, 2024 05:06

eth-p disabled auto-merge February 10, 2024 05:06

eth-p enabled auto-merge February 10, 2024 05:08

eth-p added 5 commits February 9, 2024 22:09

Add EscapeSequenceOffsetsIterator

414403b

This can be used to extract a subset of ANSI escape sequences from a string of text. I have big plans for this eventually, but for now, it'll be used to strip OSC before printing.

Add regression test for sharkdp#2541

3d04699

More specifically, the test ensures that OSC sequences don't end up wrapping the line.

Add tests for re-emitting SGR sequences on wrap

2d46d54

Add EscapeSequenceIterator

6b9b085

This is an iterator for escape sequences, using EscapeSequenceOffsetsIterator for the underlying parsing of individual escape sequences.

eth-p added 4 commits February 9, 2024 22:09

Replace AnsiCodeIterator in printer.rs

165c495

This uses the new EscapeSequenceIterator, saving us a preprocessing step for each line.

Re-emit hyperlinks when wrapping lines

6549e26

Remove hyperlink when wrapping lines

1023399

Update changelog for fix of ANSI OSC sequences

61029c8

eth-p force-pushed the fix-2541 branch from 2f3bc7f to 61029c8 Compare February 10, 2024 06:10

eth-p merged commit 5a2a20a into sharkdp:master Feb 10, 2024
22 checks passed

eth-p deleted the fix-2541 branch February 10, 2024 06:20

This was referenced Feb 12, 2024

(Draft) Attempted performance improvements for interactive printer #2855

Draft

Issue handling text with escape sequences already embedded #2185

Closed

bat slow when doing a git lg and using it as the pager compared to less #1147

Open

keith-hall mentioned this pull request Jun 29, 2024

bat (and less) output containing "^G" after clickable texts (ArchLinux-Alacritty) #3013

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Treat OSC ANSI Sequences as Invisible Text & Add OSC 8 Support #2544

Treat OSC ANSI Sequences as Invisible Text & Add OSC 8 Support #2544

eth-p commented Apr 17, 2023 •

edited

Loading

eth-p commented Apr 18, 2023

eth-p commented Apr 18, 2023 •

edited

Loading

eth-p commented Apr 18, 2023 •

edited

Loading

eth-p commented Feb 8, 2024

sharkdp left a comment

eth-p commented Feb 10, 2024

Treat OSC ANSI Sequences as Invisible Text & Add OSC 8 Support #2544

Treat OSC ANSI Sequences as Invisible Text & Add OSC 8 Support #2544

Conversation

eth-p commented Apr 17, 2023 • edited Loading

eth-p commented Apr 18, 2023

eth-p commented Apr 18, 2023 • edited Loading

eth-p commented Apr 18, 2023 • edited Loading

Notes

eth-p commented Feb 8, 2024

Benchmarks: aarch64

Benchmarks: amd64

sharkdp left a comment

Choose a reason for hiding this comment

eth-p commented Feb 10, 2024

eth-p commented Apr 17, 2023 •

edited

Loading

eth-p commented Apr 18, 2023 •

edited

Loading

eth-p commented Apr 18, 2023 •

edited

Loading

Benchmarks: `aarch64`

Benchmarks: `amd64`